Led synchronization for virtual and augmented reality devices

ABSTRACT

A master clock signal for an image capture device and a light emission device is accessed. The master clock signal is divided to generate a high frequency clock signal. A frame timer is used to measure a frame time of the image capture device based on cycles of the high frequency clock signal. Based on an exposure timing signal from the image capture device, estimating an exposure start time for the image capture device; is estimated. Based on the estimated starting time, the light emission device begins emission of a positional tracking pattern at the estimated starting time and for a duration determined by the measured frame time of the image capture device.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation of U.S. patent application Ser. No.16/877,965, filed May 19, 2020, the content of which application ishereby expressly incorporated herein by reference in its entirety.

BACKGROUND

Virtual reality (“VR”) devices enable users to view and interact withvirtual environments. For example, a VR device might enable a user toexplore a virtual environment. Augmented reality (“AR”) devices enableusers to view and interact with virtual objects while simultaneouslyviewing the physical world around them. For example, an AR device mightenable a user to view the placement of virtual furniture in a real-worldroom. Devices that enable both VR and AR experiences might be referredto as mixed reality (“MR”) devices. VR devices, AR devices, and MRdevices are also commonly referred to as a near-eye device (“NED”) orhead-mounted device (HMD). The HMD may interface with a handheldcontroller and/or other handheld peripheral objects, e.g., as parts of agaming system. To make a gaming experience more immersive, the poses ofan HMD and a handheld controller may be tracked as the devices are movedthrough space by a user to provide inputs to control a user interface ofthe HMD. Such pose tracking also may be used for other computing deviceinteractions than gaming.

In order to track a user's positional movements, or the movements of aseparate controller for the HMD, some HMD devices utilize opticalsensors such as cameras that detect LED markers that are illuminated inthe vicinity of the HMD. When using such LED markers, it may benecessary to synchronize timing between LED emission and cameraexposure. In particular, in 6 degree of freedom (DoF) AR/VR controllerswith LED constellation-based tracking systems, the timingsynchronization between the LED emitting time and the camera exposurewindow is important for image quality, accuracy, and power consumption.Bluetooth classic or Bluetooth Low Energy (BLE) typically have 10 ms orlonger packet latencies and uncertainties that are not practical for LEDemitting time control. Additionally, the camera module and Bluetoothchip on the HMD may run from different clock sources which can result inclock drift. These issues can make LED and camera timing synchronizationa challenging task.

Inaccurate synchronization can lead to poor image quality and accuracy,which can lead to issues with controller tracking and thus negativelyimpact the user experience. Additionally, inaccurate synchronization canlead to unnecessary use of computing resources and excessive powerconsumption.

It is with respect to these and potentially other considerations thatthe disclosure made herein is presented.

SUMMARY

When LED optical synchronization mechanisms are implemented, such asthat used in some 6 DoF VR controllers, the LED emitting time may be setmuch longer than the camera exposure time to ensure proper imagecapture. Because there is no timing synchronization between the HMD and6 DoF controller, the optical image synchronization accuracy may belimited. In order to reduce excessive LED emit times, it is important toactivate the LEDs close to the actual time of camera exposure. In manysystems, due to uncertainties in the camera exposure times, the LED emittime can be much longer than the camera exposure time in order to ensurethat the camera can capture the LED constellation. This can result inincreased power consumption.

Additionally, it is desirable to reduce the camera exposure time so thatthe capture image can be more clear, especially when the camera orcontroller is in motion. For an HMD, optic synchronization requires aline-of-sight view so the HMD should have a clear view of thecontroller. Furthermore, whenever the controller moves outside of thecamera field of view (FOV), the controller must enter re-initializationmode. During resynchronization, the LEDs typically need to be set tofully ON and with increased or maximum brightness in order to enableimage capture and enter re-synchronization, which also increases powerconsumption.

Communications between the controller and HMD is typically wireless,which in many cases use Bluetooth. On the HMD, when a high frequencyclock is used, the number of clock cycles for each camera frame(typically 90 Hz) needs to be counted to track camera frame times.Additionally, the start time of the LED cycle needs to be synchronizedwith the HMD.

Technologies are disclosed herein for implementing a Bluetooth physicallayer wireless synchronization mechanism that does not require changesto existing Bluetooth physical and other protocol layers. The LED timingadjustment may be implemented at the nanosecond level to set the LED ONstart time and ON duration for accurate timing control. As a result, thedescribed systems and devices may save power and generate sharper andreduced blur images for improving controller constellation trackingaccuracy, even when the devices are in motion. Once initialsynchronization is established, the synchronization may be maintaineduntil the system is powered off, even when the controller moves in andout of the camera FOV since the described wireless synchronizationmechanism does not require line of sight between the HMD camera andcontroller LED.

The described technologies can enable the reduction of erroneous inputsat AR and VR devices and reduce the utilization of computing resourcessuch as processor cycles and battery power. Technical benefits otherthan those specifically described herein might also be realized throughimplementations of the disclosed technologies.

In order to provide the technical benefits described above, andpotentially others, the present disclosure includes implementation ofphase lock loop (PLL) based clock multipliers to boost the master clock.An LED frame timer may be implemented to measure camera frame times tothe desired accuracy. A wireless digital phase lock loop may beimplemented for camera frame and LED timing synchronization by addingaveraged camera and Bluetooth clock drift error into the LED FrameCounter to ensure that the synchronization error is within a desiredrange.

A Bluetooth RF RX/TX switching control pin or other RF detection circuitmay be used as a Bluetooth synchronization reference. In one embodiment,two frame timers may be used to store the previous frame time andcurrent frame time to generate a frame time error. The averaged frametime errors may be added to the current frame timer. In an embodiment, ahardware frame counter may be used to count the frame time and generatean accurate frame trigger signal.

An LED offset estimator may be used for LED initial start timeestimation. A hardware LED offset counter and a LED duration counter maybe used to control the LED start time and duration. The frame time errorcan be calculated on the HMD as well. In this case, the HMD may send theframe time once and then send the frame error thereafter.

The HMD camera may use an exposed strobe pin or other framesynchronization signal for camera frame and LED synchronization controlbetween the HMD and controller. A parallel or serial controller with anenable pin and a pulse width modulation (PWM) input pin may be used tocontrol LED constellation on/off and brightness. The brightness may becontrolled by the HMD in a closed loop fashion. The HMD may command thecontroller to increase or decrease the brightness when the HMDdetermines that the LED constellation is too dim or too bright due tochanges in the distance between the HMD and controller. Many cameradevices have a strobe signal which can be used to track the camera starttime. To find the LED start time, in one embodiment a binary searchprocess may be implemented. In an embodiment, the LED may be turned onat the beginning of the first cycle of transmit time, and the HMD maydetermine if the LED can be detected. If the LED is not detected, thenit can be determined that the LED is not activated during the first halfof the cycle, and during the next cycle, the second half of the cyclecan be assumed. The process can continue with the time period divided inhalf, and the search can continue in this fashion until the start timecan be determined to a desired accuracy.

In some embodiments, if the same clock source is used for the HMD andcontroller, then the clock source need not be divided as there is noclock drift, while the other techniques can be employed to improve powerconsumption. In such embodiments, the PLL, hardware counters, and camerastrobe signal, as well as the camera and Bluetooth clock drifting errorestimation and correction functionality may be optional. The describedtechniques can be implemented in any situation in which the precisetiming between master and slave devices is desired, such as in a timedivision duplex (TDD) wireless communication system that uses a RF RX/TXswitching signal exposed on the HMD and 6DoF controller. While theexamples described herein are in the context of a Bluetooth mastersignal, the described principles may be applied to other industrial orproprietary radio technology-based controllers other than Bluetooth.

It should be appreciated that the subject matter described briefly aboveand in further detail below can be implemented as a computer-implementedmethod, a computer-controlled apparatus or device, a computing system,or an article of manufacture, such as a computer storage medium. Whilethe subject matter described herein is presented in the general contextof program modules that execute on one or more computing devices, thoseskilled in the art will recognize that other implementations can beperformed in combination with other types of program modules. Generally,program modules include routines, programs, components, data structures,and other types of structures that perform particular tasks or implementparticular abstract data types.

Those skilled in the art will also appreciate that aspects of thesubject matter described herein can be practiced on or in conjunctionwith other computer system configurations beyond those specificallydescribed herein, including multiprocessor systems, microprocessor-basedor programmable consumer electronics, minicomputers, mainframecomputers, hand-held computers, smartphones, personal digitalassistants, smart watches, e-readers, tablet computing devices,special-purposed hardware devices, network appliances, and the like.

Features and technical benefits other than those explicitly describedabove will be apparent from a reading of the following DetailedDescription and a review of the associated drawings. This Summary isprovided to introduce a selection of concepts in a simplified form thatare further described below in the Detailed Description. This Summary isnot intended to identify key features or essential features of theclaimed subject matter, nor is it intended that this Summary be used tolimit the scope of the claimed subject matter. Furthermore, the claimedsubject matter is not limited to implementations that solve any or alldisadvantages noted in any part of this disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a computing device diagram showing aspects of a VR device andcontroller that can be utilized to implement the various configurationsdisclosed herein;

FIG. 2 is a diagram showing aspects of the configuration and operationof a device that can be utilized to implement the various configurationsdisclosed herein;

FIG. 3 is a diagram showing aspects of the configuration and operationof a device that can be utilized to implement the various configurationsdisclosed herein;

FIG. 4 is a diagram showing aspects of the configuration and operationof a device that can be utilized to implement the various configurationsdisclosed herein;

FIG. 5 is a diagram showing aspects of the configuration and operationof a device that can be utilized to implement the various configurationsdisclosed herein;

FIG. 6 is a diagram showing aspects of the configuration and operationof a device that can be utilized to implement the various configurationsdisclosed herein;

FIG. 7 is a diagram showing aspects of the configuration and operationof a device that can be utilized to implement the various configurationsdisclosed herein;

FIG. 8 is a flow diagram showing in example process that may beimplemented in various configurations disclosed herein;

FIG. 9 is a timing diagram showing aspects of synchronization asdisclosed herein;

FIG. 10 is a flowchart depicting an example procedure forsynchronization in accordance with the present disclosure;

FIG. 11 is a flowchart depicting an example procedure forsynchronization in accordance with the present disclosure;

FIG. 12 is a device diagram showing aspects of an HMD device that can beutilized to implement the various configurations disclosed herein.

DETAILED DESCRIPTION

The disclosed examples may utilize cameras, such as a stereo cameraarrangement, on an HMD for tracking a handheld object, such as a gamecontroller. In 6DoF controllers with an LED (visible or IR)constellation-based tracking mechanism, the timing synchronizationbetween LED emission time and exposure window of the HMD camera isimportant for image quality, accuracy, and power consumption. Bluetoothclassic or BLE typically have 10 ms or longer packet latencies, which isnot practical for LED emission time control. Additionally, the cameraand Bluetooth devices on the HMD may run from different clock sourceswhich can result in clock drift. For these reasons, LED and cameratiming synchronization can be a challenging task.

The LED emission time control on some controllers may be based on anoptical synchronization algorithm where the LED emit time is longer thanthe camera exposure time in order to ensure proper image capture.Because there is no synchronization between the HMD and controller, theoptical image synchronization accuracy may be limited. Whenever thecontroller moves outside of the camera field of view (FOV), thecontroller must enter the re-initialization mode, which requires thatall LEDs turn on with maximum brightness for re-synchronization. Thiscan result in high power consumption.

The following Detailed Description describes technologies for enablingBluetooth physical layer wireless synchronization without modifyingexisting Bluetooth physical and other protocol layers. The describedtechniques enable the LED ON time to be shorter than the camera exposuretime. In this way, LED power consumption can be reduced and blurring ofthe LED constellation image can be minimized in situations where thecontroller is in motion. In some embodiments, only the RF TX/RXswitching signal for external RF PA control may be used for timingsynchronization. This signal is available in many Bluetooth chipsets.The described embodiments may be implemented independently of anyparticular chipset. Accordingly, the master and slave device chips neednot be from the same manufacturer. Additionally, the describedprinciples may be applied to other industrial or proprietary radiotechnology-based controllers other than Bluetooth.

The Bluetooth master device typically sends out a 3.2 KHz masterwireless clock that is the time base for master and slave devicephysical layer time synchronization and packet transfer. This resolutionis typically too low for 6DoF LED timing synchronization. However, the3.2 KHz master wireless clock maintains the master and slave device insync without clock drift. The present disclosure provides a way toimplement a PLL-based clock multiplier to boost the clock from 3.2 KHzto 24 MHz or higher in order to provide accurate LED timing control atnanosecond levels.

When the camera clock and Bluetooth clock use difference clock sources,clock drift may result. In an embodiment, a digital PLL circuit andcamera strobe signal may be implemented to ensure that the HMD cameraand 6DOF controller LED remain in sync to the camera frame. Once theinitial synchronization is established, the synchronization may bemaintained even if the controller moves outside the camera FOV, as longas the Bluetooth wireless link is present.

Implementations of the disclosed subject matter may reduce theutilization of computing resources such as processor cycles and batterypower by more efficiently synchronizing the controller and LEDs,reducing the possibility that the exposure needs to be repeated and thusreducing the power used by the HMD. Technical benefits other than thosespecifically described herein might also be realized throughimplementations of the disclosed technologies.

Turning now to the figures (which might be referred to herein as a“FIG.” or “FIGS.”), additional details will be provided regarding thetechnologies disclosed herein with reference to the accompanyingdrawings that form a part hereof. The FIGS. show, by way ofillustration, specific configurations or examples. Like numeralsrepresent like or similar elements throughout the FIGS.

In the FIGS., the left-most digit(s) of a reference number identifiesthe figure in which the reference number first appears. References madeto individual items of a plurality of items can use a reference numberwith another number included within a parenthetical (and/or a letterwithout a parenthetical) to refer to each individual item. Genericreferences to the items might use the specific reference number withoutthe sequence of letters. The drawings are not drawn to scale.

FIG. 1 is a computing device diagram showing aspects of theconfiguration and operation of a VR device 100 that can be utilized toimplement the various technologies disclosed herein. The VR device 100,alone or in combination with one or more other devices (e.g. a localcomputer or one or more remotely-located server computers), might form asystem that performs or otherwise implements the various processes andtechniques described herein.

In the configuration shown in FIG. 1 , the VR device 100 takes the formof a wearable, head-mounted display device that is worn by a user. Itwill be understood, however, that the VR device 100 might take a varietyof different forms other than the specific configuration depicted inFIG. 1 .

As discussed in greater detail below, the technologies disclosed hereincan also be utilized with AR devices, such as the AR device 500 shown inFIG. 5 and described below. Although the configurations disclosed hereinare discussed primarily in the context of AR and VR devices, it is to beappreciated that the technologies disclosed herein can also be utilizedwith MR devices and other types of devices that include at least onecamera such as, but not limited to, smartphones, video game systems,tablet computing devices, and smartwatches.

The VR device 100 includes one or more display panels (not shown in FIG.1 ) that display computer generated (“CG”) graphics. For example, the VRdevice 100 might include a right-eye display panel for right-eye viewingand a left-eye display panel for left-eye viewing. A right-eye displaypanel is typically located near a right eye of the user to fully orpartially cover a field of view of the right eye, and a left-eye displaypanel is located near a left eye of the user to fully or partially covera field of view of the left eye.

In another example, a unitary display panel might extend over both theright and left eyes of a user, and provide both right-eye and left-eyeviewing via right-eye and left-eye viewing regions of the unitarydisplay panel. In each of these implementations, the ability of the VRdevice 100 to separately display different right-eye and left-eyegraphical content via right-eye and left-eye displays might be used toprovide a user of the VR device 100 with a stereoscopic viewingexperience.

The VR device 100 might include a variety of on-board sensors forming asensor subsystem 120. For example, and without limitation, the sensorsubsystem 120 might include one or more outward facing optical cameras102 (e.g., cameras located on an external surface of the VR device 100and forward facing in a viewing direction of the user), such as theoptical cameras 102A and 102B shown in FIG. 1 . At least two or morecameras may be implemented for left and right FOV. The VR device 100 canalso include one or more inward facing optical cameras (not shown inFIG. 1 ) (e.g., rearward facing toward the user and/or toward one orboth eyes of the user).

The sensor subsystem 120 can also include a variety of other sensors(not shown in FIG. 1 ) including, but not limited to, accelerometers,gyroscopes, magnetometers, environment understanding cameras, depthcameras, inward or outward facing video cameras, microphones, ambientlight sensors, and potentially other types of sensors. Data obtained bythe sensors, including the cameras 102, of the sensor subsystem 120 canbe utilized to detect the location, orientation (which might be referredto as a “pose”), and movement of the VR device 100. The location,orientation, and movement of the VR device 100 can be utilized tocompute the view of the VR environment presented to the user by the VRdevice 100.

The one or more outward facing optical cameras 102 of the VR device 100can be configured to observe the real-world environment and outputdigital images illustrating the real-world environment observed by theone or more outward facing optical cameras 102. The optical cameras 102are black and white (“B&W”) cameras in one particular configuration. Itis to be appreciated, however, that other types of cameras 102 can beutilized in other configurations such as, but not limited to,red-green-blue (“RGB”) cameras and infrared cameras.

The VR device 100 might also include a processing subsystem 130 thatincludes one or more processor devices that perform some or all of theprocesses or operations described herein, as defined by instructionsexecuted by the processing subsystem 130. Such processes or operationsmight include generating and providing image signals to the displaypanels, receiving sensory signals from sensors in the sensor subsystem120 such as the cameras 102, enacting control strategies and proceduresresponsive to those sensory signals. Other computing systems, such aslocal or remote computing systems might also perform some or all of thecomputational tasks disclosed herein.

The VR device 100 might also include an on-board data storage subsystem140 that includes one or more memory devices storing computer-executableinstructions (e.g., software and/or firmware) executable by theprocessing subsystem 130, and might additionally hold other suitabletypes of data. The VR device 100 might also include a communicationssubsystem 150 supporting wired and/or wireless communications withremote devices (i.e., off-board devices) over a communications network(not shown in FIG. 1 ). As an example, the communication subsystem 150might be configured to wirelessly send or receive a video stream, audiostream, coordinate information, virtual object descriptions, and/orother information from remote devices to render virtual objects andtextures on the integrated displays of the VR device 100. Examples of VRdevices include, but are not limited to, the HTC VIVE VR device and theOCULUS RIFT VR device.

The controller 160 may be a controller, for example, for a video gamesystem. The controller 160 may, in some examples, include an onboardprocessor, storage system, and communication system. In other examples,the controller 160 may include lights to assist in optical tracking ofthe controller 160. The controller 160 may also include one or moreinput controls, such as a button, trigger, joystick, directional pad,touch screen, etc. The controller 160 may comprise an inertialmeasurement unit (IMU) (which may include an accelerometer, gyroscope,magnetometer, and/or other suitable sensors) that provides outputrelated to changes in position and orientation of the controller 160. Insome examples, the VR device 100 may also include an IMU to help trackchanges in the HMD pose in the environment.

The controller 160 may include a plurality of light sources distributedover a surface of the controller 160. FIG. 1 shows the controller 160having a plurality of light sources 170 distributed along both anexterior and interior of a ring-like structure of the controller 160.The light sources may be configured to form patterns of light(“constellations”) in image data acquired by the camera or cameras ofthe HMD, such that a pose of the controller 160 may be determined froman image capturing the controller. The light sources may take anysuitable form, such as light-emitting diodes (LEDs) that emit visiblelight for detection via a visible light camera or cameras on the HMD.Infrared light sources also may be used. In some examples, the camera(s)on the HMD may be configured to filter wavelengths of light other thanthose emitted by the LEDs to reduce noise levels in the image data. Thecontroller 160 may have any suitable number and arrangement of lightsources.

In some embodiments, the VR device 100 and controller 160 may beconfigured to communicate with one another directly, such as via awireless network connecting utilizing a Bluetooth communication protocolor other suitable wireless communication protocol. In other embodiments,an optional host computing device (not shown) may communicate with theVR device 100 and the controller 160 to receive data from VR device 100and controller 160 (including image data acquired by the VR device 100and IMU data from the controller 160), to process such data, and to sendcontrol signals to these devices. It will be understood that whiledisclosed herein in the context of handheld controllers, any suitablehandheld or movable object may be tracked, including objects without anyinternal computing system or IMU, where such objects include a pluralityof light sources detectable as a constellation in image data. Likewise,the specific communication channels shown in the FIGS. are presented forthe purpose of example, and any other suitable communications channels,wired and/or wireless, may be used for HMD-to-host communication,host-to-controller communication, and/or HMD-to-controllercommunication.

A relative-to-HMD pose (position and orientation) of the controller 160may be estimated by tracking the positions of light from the lightsources on the controller 160 using the one or more cameras on the VRdevice 100. Data from the IMU on the controller 160 can further informtracking, such as when the light sources might be occluded from view.However, the image data provides a pose of the controller 160 relativeto the VR device 100, while the IMU senses the forces proportional tothe change in position and orientation in its body frame relative to theinertial frame. Thus, to combine the optical pose from image data withthe IMU data to provide the actual pose of the controller 160 relativeto the world, processing may be used to “fuse” the IMU data with theoptical data by mapping the optical data to world coordinates.

FIG. 2 through FIG. 5 shows aspects of a system diagram of an AR/VR HMD(Head Mount Device) and 6 DoF controller synchronization with Bluetoothmaster clock.

Referring to FIG. 2 , illustrated is an HMD MCU 210, which may be amulti-media micro-controller with 2 to 4 head tracking cameras for headtracking and 6 DoF controller constellation tracking. Also illustratedis a frame timer 220 which may be part of a digital PLL configured tosynchronize a camera frame with a 24 MHz synchronized clock that isphase locked to the 3.2 KHz Bluetooth master clock. The frame timer 220measures the 90 Hz camera frame time with a 24 MHz sampling rate. Themeasured time is sent to the 6DoF slave controller device for LED timingsynchronization and drift correction.

FIG. 2 illustrates head tracking cameras 230 with a flash strobe signal235. In some implementations, there may be 2-4 or more cameras for HMDhead and controller tracking. The cameras may be synchronized from thesame clock source. One of the cameras may expose the flash strobe signal235 as the camera time synchronization base at 60/90 MHz or higher. Thestart time and duration of the strobe signal may be programmable.

Referring to FIG. 3 , illustrated is a Bluetooth master antenna 310.Also illustrated is Bluetooth master main clock 320, shown as BT MCLK1(24 MHz). The Bluetooth master main clock 320 is the clock reference forthe master Bluetooth chip. Also illustrated in FIG. 3 is the Bluetoothmaster module 330, which is the Bluetooth master controller on the HMDwith the TX/RX switching signal exposed on the chip.

FIG. 3 further illustrates the master clock generator 340, whichgenerates the 3.2 KHz master clock 360 for the Bluetooth master-slavephysical layer timing reference. Also illustrated is the clockmultiplier 350 which, in one embodiment, multiplies the master clock by7500 to generate a 24 MHz synced clock for the frame timer input clock.In one implementation, the Cirrus Logic CS2000-OTP may be used. Itshould be understood that the master clock may be multiple by othervalues depending on the specific implementation.

Referring to FIG. 4 , illustrated is the LED driver 410 which drives oneor more LED strings 420 with an enable pin 412 and PWM input 414 for LEDON/OFF and brightness control, respectively. Also illustrated iscontroller MCU 430 which may be a micro-controller for 6DoF controllerLED timing and Bluetooth communication. LED string 420 comprises one ormore serial or parallel LED strings configured to generate theconstellation for 6DoF controller tracking.

FIG. 4 further illustrates frame time synchronizer 440 which generatesthe camera frame time from the frame time packet received from themaster controller 430. The frame time synchronizer 440 compares thecurrent and newer frame timer to estimate clock drift that is averagedto the frame timer before the next frame time pack is received during afixed period. In this way, the frame time synchronizer 440 compensatesfor camera clock drift frame by frame without relying on the Bluetoothpacket latency which could be as high as hundreds of milliseconds.

FIG. 4 further illustrates LED offset estimator 450 which may beconfigured to estimate the LED ON timing offset with reference to theframe start point for initial synchronization. Also illustrated is theLED offset counter 460 which stores the LED ON timing offset. The LEDoffset counter 460 may be adjusted to an accuracy of 1/24 MHz=41.6 ns,in some embodiments. FIG. 4 further illustrates the LED duration counter470 which may be used to control LED ON time and may be adjustable. Ashorter LED duration may enable a sharp and stable image when thecontroller is in motion.

Referring to FIG. 5 , illustrates the device Bluetooth antenna 510. Alsoillustrated is the Bluetooth slave module 520 which may be a Bluetoothslave device controller on the 6DoF controller with TX/RX switchingsignal exposed for master and slave synchronization. FIG. 5 furtherillustrates the master clock recovery function 530 that recovers the 3.2KHz master clock as one of the clock multiplier inputs, which isinherent in the Bluetooth physical layer as a timing reference and RX/TXswitching control. At a 1.5 m distance, the master clock and recoveredmaster clock may have a 5 ns difference due to the TDD timing adjustmentin the physical layer. The difference can be ignored for a 41.6 nstiming resolution without noticeable differences in LED timing control.

FIG. 5 further illustrates the clock multiplier 540 which may be thesame as clock multiplier 350 in FIG. 3 and used for master clockmultiplication to generate the 24 MHz synced clock as a frame timereference on the slave device. Also illustrated is Bluetooth MCK2 (24MHz) 550 which is the Bluetooth slave main clock that is the clockreference for the slave Bluetooth chip. The Bluetooth slave main clockhas a clock that drifts compared to the master main clock because themaster and slave use different clock sources. However, the output of theslave clock multiplier 540 is phase locked to the 3.2 KHz master clockto overcome the clock drift.

Referring to FIG. 6 , illustrated are various components on the HMDaccording to some embodiments. Illustrated is a PLL based ClockMultiplier 600 that takes a 24 MHz (BT MCLK) f1 clock as one of twoinputs. The other input is the 3.2 KHz BT master clock. The output f3 is24 MHz but locked to the master clock phase which allows for 24 MHz f3clock recovery on the controller side without clock drift. Alsoillustrated is HMD MCU timer/counter 610 with an external event triggerthat is available on many micro-controllers. The 24-bit timer/counterinput is phase locked to the 24 MHz f3 clock. The trigger event is thecamera strobe signal 630, which is an accurate camera exposure timingreference. The camera strobe signal 630 may be used to measure thecamera frame period and send the camera frame period to the controllerMCU for the LED timing reference.

For the 10 ppm clock source, the maximum measurement error per frame is0.22 us. The error can accumulate frame by frame due to the camera andBT clock having different sources. This error can be eliminated by usingthe same 24 MHz clock source for the camera and BT on the HMD side or bysending the camera frame time measured by the 24 MHz clock f3 clock tothe controller for periodic correction, such as every 100 frames.

The error ΔFT may be averaged to ΔFTi=[ΔFT/n], i=1˜100, n=100 forexample. ΔFTi is an integer and the remainder of ΔFT/n may be added tothe current frame counter, one per frame. In this way, the timedifference between adjacent frames is no more than 41.6 ns.

FIG. 6 further illustrates the HMD head and controller tracking cameras640 with the strobe signal exposed. The cameras 640 may be synchronizedfrom the same clock source. Camera sensors typically have a strobesignal for flash light control. In an embodiment, at least one of thecameras 640 has its strobe signal exposed. Although the framesynchronization signal may be used for the same purpose, the strobesignal is programmable and more flexible.

FIG. 7 illustrates various components on the controller according tosome embodiments. Illustrated is clock multiplier 710 which may be thesame clock multiplier part as clock multiplier 600 of FIG. 6 . The clockmultiplier 710 may be used to generate the 24 MHz f3 clock that is phaselocked to the 3.2 KHz master clock for LED timing control. The 24 MHz f1and f1′ may be slightly different due to their being from a differentsource, but both are phase locked to the 3.2 MHz BT master clock.

Also illustrated in FIG. 7 are two 24-bit registers holding a new frametimer (NFT) 720 and current frame timer (CFT) 730. The NFT 720 may beconfigured to hold the camera frame time received from the HMD in thecamera frame time Bluetooth packet 725. The CFT 730 may hold the frametime compensated by the averaged frame time error resulting from theclock drift ΔFT. The CFT 730 may also include a hardware frame counter740 configured to generate a 90 Hz frame start trigger signal whenevercounter underflow occurs. The block 750 may provide a wireless digitalphase lock loop to ensure that the camera frame and LED control timingis synchronized until the HMD or 6DoF controller is powered off.

Block 760 includes a software LED offset estimator (LOE) 762 and twohardware counters: LED offset counter (LOC) 764 and LED duration counter(LDC) 766 at a resolution of 41.6 ns. The LOE 762 may estimate the LEDstarting time referenced to the frame trigger signal in the initialsynchronization process. This time value may be stored in the LOC 764.When the LOC 764 underflows, it may trigger the LED to turn the LED on.At the same time, the LDC 766 may start counting down and turn the LEDoff when it underflows. The LDC 766 may be set by software. The LDC 766may be the same as or shorter than the camera exposure time to allow fora sharp image to be captured when the 6DoF controller is in motion. TheLED duration time may be programmable and may be determined by imagesharpness requirements, the LED driving current, the camera sensorsensitivity, etc. FIG. 7 also illustrates an LED drive 770 with theenable pin and PWM for LED ON/OFF and brightness control, respectively.The brightness control data may be feedback from the HMD via the BT datapacket.

FIG. 8 illustrates a procedure that may be used to estimate LED offsetthrough a handshake between the HMD and Controller. A binary searchalgorithm may be employed to determine the LED timing offset. Theprocedure may set LDC=CFT. The HMD initialization is illustrated inblock 800. Block 802 illustrates that a Start command may be send to thecontroller. In block 804, it is determined if the initialization requesthas been acknowledged. If the request has not been acknowledged, thenblock 802 may be repeated. If the request has been acknowledged, thenblock 806 illustrates determining if the LED image has been captured. Ifthe LED image has not been captured, then an LED_NAK 810 may be sent. Ifthe LED image has been captured, then an LED_ACK 808 may be sent.

Block 812 illustrates determining if initialization has been completed.If initialization has not been completed, then the procedure returns toblock 806. If initialization has been completed, then HMD initializationis completed 814.

Block 820 illustrates initialization of the controller. In block 822, itis determined if the initialization has started. If the initializationhas not been started, then block 822 may be repeated. If theinitialization has started, then block 824 illustrates sending anINI_ACK. Block 826 illustrates setting the LED target duration.

Block 828 illustrates setting the LED Offset Counter (LOC) to 0. Block830 illustrates setting the LED Duration Counter (LDC) to the CameraFrame Time (CFT). Block 832 illustrates setting the LED On/Off based onthe LED Offset Counter and LED Duration Counter 832. Block 834illustrates determining if the LED_ACK or LED_NAK has been received. Ifthe LED_ACK or LED_NAK has not been received, then block 834 isrepeated. If the LED_ACK or LED_NAK has been received, then the LEDDuration Counter may be divided by 2 as illustrated in block 836.

Block 838 illustrates determining if the LED_ACK or LED_NAK has beenreceived. If the LED_ACK has been received, then block 840 illustratesLED Offset Counter remaining at the current value. If the LED_NAK hasbeen received, then block 842 illustrates adding the LED DurationCounter to the LED Offset Counter.

Block 844 illustrates determining if the LED Duration Counter is lessthan the LED Duration Counter target duration. If the LED DurationCounter is not less than the LED Duration Counter target duration, thenthe procedure returns to block 832. If the LED Duration Counter is lessthan the LED Duration Counter target duration, then the LED DurationCounter is set to the LED target duration as shown in block 846. Block848 illustrates sending the initialization complete message. Operation850 completes the procedure.

FIG. 9 illustrates a camera and LED initial synchronization example withLDC= 1/64 and CFT=174 us (90 pfs). The timing sequence 910 illustratesthe camera exposure time and the camera frame time. At step 0, the LEDOffset Counter is set to 0 and the LED Duration Counter is set to thecamera frame time. At each successive step, the LED Offset Counter isupdated until the LED On Time corresponds to the camera exposureduration and start time.

FIGS. 10 and 11 are flow diagrams showing aspects of a routine disclosedherein for synchronizing an exposure time of an image capture device fora virtual reality/augmented reality (VR/AR) headset and an emission timeof a communicatively coupled light emission device of a controller, theVR/AR headset configured to track positional movement of the controllerbased on tracking light emitted by the light emission device.

It should be understood by those of ordinary skill in the art that theoperations of the methods disclosed herein are not necessarily presentedin any particular order and that performance of some or all of theoperations in an alternative order(s) is possible and is contemplated.The operations have been presented in the demonstrated order for ease ofdescription and illustration. Operations may be added, omitted,performed together, and/or performed simultaneously, without departingfrom the scope of the appended claims.

It should also be understood that the illustrated methods can end at anytime and need not be performed in their entireties. Some or alloperations of the methods, and/or substantially equivalent operations,can be performed by execution of computer-readable instructions includedon a computer-storage media, as defined herein. The term“computer-readable instructions,” and variants thereof, as used in thedescription and claims, is used expansively herein to include routines,applications, application modules, program modules, programs,components, data structures, algorithms, and the like. Computer-readableinstructions can be implemented on various system configurations,including single-processor or multiprocessor systems, minicomputers,mainframe computers, personal computers, hand-held computing devices,microprocessor-based, programmable consumer electronics, combinationsthereof, and the like. Although the example routine described below isoperating on a computing device, it can be appreciated that this routinecan be performed on any computing system which may include a number ofcomputers working in concert to perform the operations disclosed herein.

Thus, it should be appreciated that the logical operations describedherein are implemented (1) as a sequence of computer implemented acts orprogram modules running on a computing system such as those describedherein and/or (2) as interconnected machine logic circuits or circuitmodules within the computing system. The implementation is a matter ofchoice dependent on the performance and other requirements of thecomputing system. Accordingly, the logical operations may be implementedin software, in firmware, in special purpose digital logic, and anycombination thereof.

Referring to FIG. 10 , operation 1001 illustrates accessing a Bluetoothmaster clock signal for the image capture device and the light emissiondevice.

Operation 1001 may be followed by operation 1003. Operation 1003illustrates dividing the Bluetooth master clock signal to generate ahigher frequency clock signal.

Operation 1003 may be followed by operation 1005. Operation 1005illustrates using a frame timer to measure a frame time of the imagecapture device based on cycles of the higher frequency clock signal.

Operation 1005 may be followed by operation 1007. Operation 1007illustrates based on an exposure timing signal from the image capturedevice and using clock cycles of the higher frequency clock signal,estimating an exposure start time for the image capture device.

Operation 1007 may be followed by operation 1009. Operation 1009illustrates synchronizing the exposure time of the image capture deviceof the VR/AR headset and the emission time of the light emission deviceof the controller based on the estimated starting time and the measuredframe time.

Operation 1009 may be followed by operation 1011. Operation 1011illustrates causing the light emission device to begin emission of apositional tracking pattern at the estimated starting time and for aduration determined by the measured frame time of the image capturedevice. In an embodiment, the VR/AR headset is configured to trackpositional movement of the controller based on synchronization of thetracking light emitted by the light emission device and the exposuretime of the image capture device.

In an embodiment, an offset estimator is used to estimate an initialexposure starting time for the image capture device.

In an embodiment, the dividing is performed using one or more phase lockloop (PLL) multipliers.

In an embodiment, averaged clock drift errors for the image capturedevice are added to the frame timer.

In an embodiment, two frame timers are used to store a previous frametime and a current frame time.

In an embodiment, the two frame timers are an LED offset counter and anLED duration counter.

In an embodiment, a RX/TX switching control pin is used as asynchronization reference.

In an embodiment, the exposure timing signal is a strobe sync signal.

In an embodiment, the estimating the starting time for the image capturedevice is performed using a binary search algorithm.

Turning now to FIG. 11 , illustrated is an example operational procedurefor synchronizing an exposure time of an image capture device for avirtual reality/augmented reality (VR/AR) headset and an emission timeof a communicatively coupled light emission device of a controller, theVR/AR headset configured to track positional movement of the controllerbased on tracking light emitted by the light emission device, inaccordance with the present disclosure. The operational procedure may beimplemented in a system comprising one or more computing devices.Referring to FIG. 11 , operation 1101 illustrates receiving a masterclock signal for the image capture device and the light emission device.

Operation 1101 may be followed by operation 1103. Operation 1103illustrates dividing the master clock signal to generate a highfrequency clock signal.

Operation 1103 may be followed by operation 1105. Operation 1105illustrates measuring a frame time of the image capture device based oncycles of the high frequency clock signal.

Operation 1105 may be followed by operation 1107. Operation 1107illustrates based on a timing signal from the image capture device,estimating a start time for the image capture device.

Operation 1107 may be followed by operation 1109. Operation 1109illustrates based on the estimated start time, causing the lightemission device to begin emission of a positional tracking pattern atthe estimated start time and for a duration determined by the measuredframe time of the image capture device.

In an embodiment, the dividing is performed using one or more phase lockloop (PLL) multipliers.

In an embodiment, averaged clock drift errors for the image capturedevice are added to the frame time.

In an embodiment, the estimating the starting time for the image capturedevice is performed using a binary search algorithm.

In an embodiment, the master clock signal is a Bluetooth timing signal.

FIG. 12 is a computing device diagram showing aspects of theconfiguration of an AR device 1200 that can be utilized to implement thevarious configurations disclosed herein. As described briefly above, ARdevices superimpose CG images over a user's view of a real-worldenvironment. For example, an AR device 1200 such as that shown in FIG.12 might generate composite views to enable a user to visually perceivea CG image superimposed over a visually perceived physical object thatexists within a real-world environment.

In the example shown in FIG. 12 , an optical system 1202 includes anillumination engine 1204 to generate electromagnetic (“EM”) radiation,including some or all of the visible-light portion of the EM spectrum.In this example, the optical system 1202 further includes an opticalassembly 1206 that is positioned to receive the EM radiation from theillumination engine 1204 and to direct the EM radiation (or individualbandwidths thereof) along one or more predetermined optical paths. Forexample, the illumination engine 1204 may emit the EM radiation into theoptical assembly 1206.

In some instances, a user experience is dependent on the AR device 1200accurately identifying characteristics of a physical object or plane andthen generating the CG image in accordance with these identifiedcharacteristics. For example, suppose that the AR device 1200 isprogrammed to generate a user perception that a virtual gaming characteris running towards and ultimately jumping over a real-world structure.To achieve this user perception, the AR device 1200 might obtaindetailed data defining features of the real-world terrain around the ARdevice 1200.

In some examples, the AR device 1200 utilizes an optical system 1202 togenerate a composite view (e.g., from a perspective of a user that iswearing the AR device 1200) that includes both one or more CG images anda view of at least a portion of the real-world environment. For example,the optical system 1202 might utilize various technologies such as, forexample, AR technologies to generate composite views that include CGimages superimposed over a real-world view. As such, the optical system1202 might be configured to generate CG images via an optical assembly1206 that includes a display panel 1214.

In the illustrated example, the display panel includes separate righteye and left eye transparent display panels, labeled 1214R and 1214L,respectively. In some examples, the display panel 1214 includes a singletransparent display panel that is viewable with both eyes or a singletransparent display panel that is viewable by a single eye only.Therefore, it can be appreciated that the techniques described hereinmight be deployed within a single-eye device (e.g. the GOOGLE GLASS ARdevice) and within a dual-eye device (e.g. the MICROSOFT HOLOLENS ARdevice).

Light received from the real-world environment passes through thesee-through display panel 1214 to the eye or eyes of the user. Graphicalcontent displayed by right-eye and left-eye display panels, ifconfigured as see-through display panels, might be used to visuallyaugment or otherwise modify the real-world environment viewed by theuser through the see-through display panels 1214. In this configuration,the user is able to view virtual objects that do not exist within thereal-world environment at the same time that the user views physicalobjects within the real-world environment. This creates an illusion orappearance that the virtual objects are physical objects or physicallypresent light-based effects located within the real-world environment.

In some examples, the display panel 1214 is a waveguide display thatincludes one or more diffractive optical elements (“DOEs”) forin-coupling incident light into the waveguide, expanding the incidentlight in one or more directions for exit pupil expansion, and/orout-coupling the incident light out of the waveguide (e.g., toward auser's eye). In some examples, the AR device 1200 further includes anadditional see-through optical component, shown in FIG. 12 in the formof a transparent veil 1216 positioned between the real-world environmentand the display panel 1214. It can be appreciated that the transparentveil 1216 might be included in the AR device 1200 for purely aestheticand/or protective purposes.

The AR device 1200 might further include various other components (notall of which are shown in FIG. 12 ), for example, front-facing cameras102 (e.g. RGB, B&W, or IR cameras), speakers, microphones,accelerometers, gyroscopes, magnetometers, temperature sensors, touchsensors, biometric sensors, other image sensors, energy-storagecomponents (e.g. battery), a communication facility, a globalpositioning system (“GPS”) a receiver and, potentially, other types ofsensors. Data obtained from one or more sensors 1208, some of which areidentified above, can be utilized to determine the orientation,location, and movement of the AR device 1200.

In the illustrated example, the AR device 1200 includes one or morelogic devices and one or more computer memory devices storinginstructions executable by the logic device(s) to implement thefunctionality disclosed herein. In particular, a controller 1218 caninclude one or more processing units 1220, one or more computer-readablemedia 1222 for storing an operating system 1224, other programs (such asa LED synchronization module 1226 configured to in the manner disclosedherein), and data.

In some implementations, the AR device 1200 (and MR devices) isconfigured to analyze data obtained by the sensors 1208 to performfeature-based tracking of an orientation of the AR device 1200. Forexample, in a scenario in which the object data includes an indicationof a stationary object within the real-world environment (e.g., atable), the AR device 1200 might monitor a position of the stationaryobject within a terrain-mapping field-of-view (“FOV”). Then, based onchanges in the position of the stationary object within theterrain-mapping FOV and a depth of the stationary object from the ARdevice 1200, the AR device 1200 might calculate changes in theorientation of the AR device 1200.

It can be appreciated that these feature-based tracking techniques mightbe used to monitor changes in the orientation of the AR device 1200 forthe purpose of monitoring an orientation of a user's head (e.g., underthe presumption that the AR device 1200 is being properly worn by auser). The computed orientation of the AR device 1200 can be utilized invarious ways, some of which have been described above.

The processing unit(s) 1220, can represent, for example, a centralprocessing unit (“CPU”) −type processing unit, a graphics processingunit (“GPU”) −type processing unit, a field-programmable gate array(“FPGA)”, one or more digital signal processors (“DSPs”), or otherhardware logic components that might, in some instances, be driven by aCPU. For example, and without limitation, illustrative types of hardwarelogic components that can be used include Application-SpecificIntegrated Circuits (“ASICs”), Application-Specific Standard Products(“ASSPs”), System-on-a-Chip Systems (“SOCs”), Complex Programmable LogicDevices (“CPLDs”), etc.

As used herein, computer-readable media, such as computer-readable media1222, can store instructions executable by the processing unit(s) 1220,such as instructions which, when executed, provide LED synchronizationin the manner disclosed herein. Computer-readable media can also storeinstructions executable by external processing units such as by anexternal CPU, an external GPU, and/or executable by an externalaccelerator, such as an FPGA type accelerator, a DSP type accelerator,or any other internal or external accelerator. In various examples, atleast one CPU, GPU, and/or accelerator is incorporated in a computingdevice, while in some examples one or more of a CPU, GPU, and/oraccelerator is external to a computing device.

Computer-readable media can include computer storage media and/orcommunication media. Computer storage media can include one or more ofvolatile memory, nonvolatile memory, and/or other persistent and/orauxiliary computer storage media, removable and non-removable computerstorage media implemented in any method or technology for storage ofinformation such as computer-readable instructions, data structures,program modules, or other data.

Thus, computer storage media includes tangible and/or physical forms ofmedia included in a device and/or hardware component that is part of adevice or external to a device, including but not limited to randomaccess memory (“RAM”), static random-access memory (“SRAM”), dynamicrandom-access memory (“DRAM”), phase change memory (“PCM”), read-onlymemory (“ROM”), erasable programmable read-only memory (“EPROM”),electrically erasable programmable read-only memory (“EEPROM”), flashmemory, rotating media, optical cards or other optical storage media,magnetic storage, magnetic cards or other magnetic storage devices ormedia, solid-state memory devices, storage arrays, network attachedstorage, storage area networks, hosted computer storage or any otherstorage memory, storage device, and/or storage medium that can be usedto store and maintain information for access by a computing device.

In contrast to computer storage media, communication media can embodycomputer-readable instructions, data structures, program modules, orother data in a modulated data signal, such as a carrier wave, or othertransmission mechanism. As defined herein, computer storage media doesnot include communication media. That is, computer storage media doesnot include communications media consisting solely of a modulated datasignal, a carrier wave, or a propagated signal, per se.

Based on the foregoing, it should be appreciated that technologies forLED synchronization have been disclosed herein. Moreover, although thesubject matter presented herein has been described in language specificto computer structural features, methodological acts, and computerreadable media, it is to be understood that the invention defined in theappended claims is not necessarily limited to the specific features,acts, or media described herein. Rather, the specific features, acts,and media are disclosed as example forms of implementing the claims.

The subject matter described above is provided by way of illustrationonly and should not be construed as limiting. Furthermore, the claimedsubject matter is not limited to implementations that solve any or alldisadvantages noted in any part of this disclosure. Variousmodifications and changes can be made to the subject matter describedherein without following the example configurations and applicationsillustrated and described, and without departing from the true spiritand scope of the present invention, which is set forth in the claimsbelow.

What is claimed is:
 1. A method for synchronizing an exposure time of animage capture device and an emission time of a communicatively coupledlight emission device of a controller, the method comprising: accessinga Bluetooth master clock signal for the image capture device and thelight emission device; dividing the Bluetooth master clock signal togenerate a higher frequency clock signal; measuring a frame time of theimage capture device based on cycles of the higher frequency clocksignal; based on an exposure timing signal from the image capture deviceand clock cycles of the higher frequency clock signal, estimating anexposure start time for the image capture device; and synchronizing theexposure time of the image capture device and the emission time of thelight emission device of the controller based on the estimated startingtime and the measured frame time.
 2. The method of claim 1, furthercomprising using an offset estimator to estimate an initial exposurestarting time for the image capture device.
 3. The method of claim 1,wherein the dividing is performed using one or more phase lock loop(PLL) multipliers.
 4. The method of claim 3, further comprising addingaveraged clock drift errors for measuring the frame time.
 5. The methodof claim 4, further comprising storing a previous frame time and acurrent frame time.
 6. The method of claim 5, wherein the previous frametime and current frame time are stored using two frame timers comprisingan LED offset counter and an LED duration counter.
 7. The method ofclaim 1, further comprising using a RX/TX switching control pin as asynchronization reference.
 8. The method of claim 1, wherein theexposure timing signal is a strobe sync signal.
 9. The method of claim1, wherein the estimating the exposure start time is performed using abinary search algorithm.
 10. A system, comprising: an image capturedevice; a controller communicatively, the controller comprising a lightemission device; and a computer storage medium having instructionsstored thereupon which, when executed by a processor, cause the systemto: receive a master clock signal for the image capture device and thelight emission device; divide the master clock signal to generate a highfrequency clock signal; measure a frame time of the image capture devicebased on cycles of the high frequency clock signal; and based on atiming signal from the image capture device, estimate a start time forthe image capture device.
 11. The system of claim 10, wherein thedividing is performed using one or more phase lock loop (PLL)multipliers.
 12. The system of claim 10, further comprising addingaveraged clock drift errors to the frame time.
 13. The system of claim10, wherein the estimating the start time is performed using a binarysearch algorithm.
 14. The system of claim 10, wherein the master clocksignal is a Bluetooth timing signal.
 15. A controller comprising: alight emission device; wherein the controller is communicatively coupledto an image capture device; and a computer storage medium havinginstructions stored thereupon which, when executed by a processor, causethe controller to: receive a master clock signal for the controller;divide the master clock signal to generate a high frequency clocksignal; measure a frame time of the image capture device based on cyclesof the high frequency clock signal; and based on a timing signal fromthe image capture device, estimate a start time for the image capturedevice.
 16. The controller of claim 15, wherein the master clock signalis common to the controller and a VR/AR headset.
 17. The controller ofclaim 15, wherein the dividing is performed using one or more phase lockloop (PLL) multipliers.
 18. The controller of claim 15, wherein theestimating the start time for the image capture device is performedusing a binary search algorithm.
 19. The controller of claim 15, whereinthe master clock signal is a Bluetooth timing signal.
 20. The controllerof claim 15, further comprising using two frame timers to store aprevious frame time and a current frame time, wherein the two frametimers are an LED offset counter and an LED duration counter.